Scanning-laser ophthalmoscopes (SLO) are medical devices which are becoming increasingly common and increasingly useful for the visualization of the retina. A recent innovation is to combine the SLO with adaptive optics (AO) which compensates for the optical distortions of the eye to provide an unprecedented view of the retina in-vivo. Such an AO-SLO device is able to take images of a retina with a resolution in the order of microns, enabling the doctor to observe images of the retina that can resolve individual photoreceptors.
The AO-SLO can capture images of the retina continuously and therefore is often used to take video sequences of the retina, where a video sequence may be a sequence of consecutively captured still images. This enables the doctor to view the change of the retina at a microscopic scale and see, for example, the transit of blood cells through the retinal blood vessels, and conduct subsequent measurement of the velocity of blood flow.
To form the image of the retina, an AO-SLO device scans a laser in a raster pattern over a small area on the surface of the retina and detects the amount of light returned. A representative device has an oscillating mirror driven at a frame rate of approximately 60 Hz, with a resonant scanner forming scan-lines at a rate of 12 kHz and a pixel clock of 5 MHz. This device captures a video at 60 frames per second comprising images of a representative size of 400×400 pixels.
While the AO-SLO is able to compensate for aberrations of the vision-forming lens and cornea of the eye, it cannot compensate for image distortion caused by the movement of the eye. The human eye is in constant motion, and this motion cannot be controlled consciously. Therefore, in a conscious patient the eye is expected to be in motion. In general, there are two types of eye movements. One type of eye movements is called drift and consists of random and slow eye movements. Another type of eye movement is the saccade, which is a high acceleration ballistic movement that moves the eye quickly from one position to another.
When the eye is moving during the raster scanning process, many image distortions are caused, and are analogous to rolling shutter distortion in some modern digital sensors, for example those commonly used in mobile phones and portable video cameras. Eye-motion can cause compression or expansion of the image when motion is in the slow-scan direction, and shearing when motion is in the fast-scan direction. However real eye motion is typically complex and consists of random drift and saccades in multiple directions which cause complicated image distortions. In addition, saccades give very fast eye motion which can cause highly problematic distortions that often cannot be corrected, even when the exact motion of the saccade is known. A saccade can additionally cause a rapid translation of the field of view of the retina from image to image, resulting in large shifts between two neighboring images. If a saccade begins, or ends, in the middle of a scan, such distortions may change across the image, or may appear in combination.
Without any further processing, the images captured by the AO-SLO are not generally useful for doctors due to the large distortions in images that have been introduced by the eye motion. The effect of these distortions on the retinal video from the AO-SLO is to cause them to be both warped and to change location from image to image so that it is extremely difficult to see stable structures on the retina. Therefore, it is important to minimize the effect of the eye motion. One method to do this is to have the patient fixate their vision by locking their gaze at a visible marker; this reduces the wandering of the eye. However, the eye will still move even when fixated and additionally there are many patients who have poor vision and the fixation point is difficult to see or not visible at all. Also, there are automated systems which can track eye motion and compensate for some of the eye movement. However, these systems cannot compensate for all eye motion, and typically have a lag which means that there are still residual distortions that must be accounted for.
Therefore, it is desirable to perform alignment processing to reduce the effect of the eye motion and stabilize images across the captured image sequence which is robust to large and small eye motion. Alignment processing traditional consists of positioning images in the AO-SLO video sequence so that the same physical feature in one image is located at the same position in all the images of an image sequence. There are many methods to perform such alignment of AO-SLO image sequences. A typical alignment process for a sequence of AO-SLO images involves selecting a reference image from the sequence and calculating the location of image features in the reference image in the rest of the images. Once the locations of common features are known, the images can be aligned using a number of methods. One problem with this approach is that the amount of overlap between the reference image and any other image from the sequence can vary significantly, especially with rapid translations of the field of view that are commonly caused by saccades. In cases where there are substantial shifts between the images, registration performance in the non-overlapping regions is usually poor due to the lack of alignment features.
FIG. 1 shows a schematic representation of the alignment of two frames 120 and 130 with a reference frame 110. This illustration shows that, for any given pair of images from an AO-SLO sequence, there are likely to be some regions which appear in one image but not in another image. The first frame 120 has a large overlap with frame 110 and can be aligned well. However, the second frame 130 has a low overlap and there is a large area 135 for which there is no overlap with the reference frame 110. In this non-overlapping region 135 alignment information is not available, and this can lead to poor registration results. Note that the non-overlapping region 135 only contains the area which has no overlap between the reference 110 and target image 130 in an entire column. Any area where the target image 130 does not overlap some pixels of the reference image 110 but there is overlap in another part of the column is not considered to be non-overlapping. For example, the area 140 of the target frame 130 below the reference frame 110 but to the right of the non-overlap region 135 is considered to be overlapping. Given that scanning in x direction is much slower compared to scanning in y direction, the eye motion is substantially the same within each column in the image and different across columns in the image. Therefore knowledge of the eye position in the upper part of an image can be used to estimate the eye position in the lower part of an image.